Apparatus and method for generating new viewpoint image

ABSTRACT

An apparatus for generating a new viewpoint image includes: a new viewpoint image generation unit that shifts the foreground image region in the input image, based on a depth map to generate a new viewpoint image including a missing pixel region; a representative background pixel selection unit that selects background pixels located a predetermined number of pixels away from edge pixels, as representative background pixels; a texture direction map calculation unit that calculates a texture direction of each of the selected representative background pixels, to calculate a texture direction map indicating texture directions, each corresponding to one of missing pixels, based on locations of the representative background pixels and locations of the missing pixels included in the missing pixel region; and a hole-filling processing unit that fills the missing pixel region with the obtained background pixels in accordance with the texture direction map.

CROSS REFERENCE TO RELATED APPLICATION

This is an application based on and claims priority of Japanese PatentApplication No. 2011-262866 filed on Nov. 30, 2011. The entiredisclosure of the above-identified application, including thespecification, drawings and claims is incorporated herein by referencein their entirety.

FIELD

One or more exemplary embodiment disclosed herein relate generally to anapparatus and a method for generating a new viewpoint image from aninput image.

BACKGROUND

It is known that an image of a scene viewed from a new viewpoint can begenerated using a rendering technique based on a depth image (depthimage based rendering, DIBR). In other words, it is possible to generatea new viewpoint image of a scene viewed from a new viewpoint, using aninput image of a scene viewed from a predetermined viewpoint and a depthimage representing the distance between objects in the scene. This newviewpoint image can be generated based on camera geometry usinginformation on the camera position of an original viewpoint and thecamera position of a new viewpoint. An original viewpoint image (inputimage) and the generated new viewpoint image can be used, for example,for 3D display as a left eye image and a right eye image. Likewise, agroup of multi-view images can be generated. The group of the multi-viewimages can be used for multi-viewpoint 3D display.

Therefore, by using the DIBR technique, a 3D image for 3D display can begenerated using an input image that is a one-viewpoint image and itscorresponding depth map.

However, when a new viewpoint image is generated using an input image,an unknown region (hereinafter referred to also as a hole region) occursin a region corresponding to an occluded region in the input image(original viewpoint image). Here, the unknown region includes noinformation or is empty. This is because the occluded region in theoriginal viewpoint image is visualized (disoccluded) in the newviewpoint image. Therefore, in generating a new viewpoint image, it isnecessary to fill a hole region using a hole-filling method or adisocclusion method.

Here, Patent Literature 1 suggests an in-painting method based on depth,and Patent Literature 2 suggests a hole-filling method based on thedirection of image features.

CITATION LIST Patent Literature

-   [PL1] U.S. Patent Application Publication No. 2010/0238160-   [PL2] U.S. Patent Application Publication No. 2009/0016640

SUMMARY

However, conventional methods shown in the Patent Literatures 1 and 2have a problem that a clear and natural new view point image can not begenerated. This decreases the visual quality of a 3D image. In otherwords, a user would feel the 3D image unnatural.

One non-limiting and exemplary embodiment provides an apparatus and amethod for generating a new viewpoint image that can generate a clearerand more natural new viewpoint image.

In one general aspect, the techniques disclosed here feature anapparatus for generating a new viewpoint image, which generates, from aninput image, a new viewpoint image that is an image of the input imageviewed from a new viewpoint, the input image including a foregroundimage region that is an image region of an object and a background imageregion other than the foreground image region, the apparatus including:a generation unit that shifts the foreground image region in the inputimage, based on a depth map corresponding to the input image to generatea new viewpoint image including a missing pixel region; a selection unitthat selects background pixels located a predetermined number of pixelsaway from edge pixels, from among background pixels included in thebackground image region, as representative background pixels, the edgepixels indicating a boundary between the foreground image region and thebackground image region in the input image; a texture direction mapcalculation unit that calculates a texture direction of each of theselected representative background pixels, the texture direction beingan edge direction of texture in the background image region for each ofthe representative background pixels, and to calculate a texturedirection map indicating texture directions, each corresponding to oneof missing pixels, based on locations of the representative backgroundpixels and locations of the missing pixels included in the missing pixelregion; and a hole-filling processing unit that fills the missing pixelregion with the background pixels in accordance with the texturedirection map.

It should be noted that these general and specific aspects may beimplemented using a system, a method, an integrated circuit, a computerprogram or a computer-readable recording medium such as a CD-ROM, or anycombination of systems, methods, integrated circuits, computer programs,or recording media.

Additional benefits and advantages of the disclosed embodiments will beapparent from the Specification and Drawings. The benefits and/oradvantages may be individually obtained by the various embodiments andfeatures of the Specification and Drawings, which need not all beprovided in order to obtain one or more of such benefits and/oradvantages.

An apparatus and a method for generating a new viewpoint image accordingto one or more exemplary embodiments or features disclosed hereinprovide a clearer and more natural new viewpoint image.

BRIEF DESCRIPTION OF DRAWINGS

These and other objects, advantages and features of the disclosure willbecome apparent from the following description thereof taken inconjunction with the accompanying drawings, by way of non-limitingexamples of embodiments disclosed herein.

FIG. 1 is a block diagram showing a configuration example of anapparatus for generating a new viewpoint image 100 according to thefirst embodiment.

FIG. 2A is a block diagram showing a detailed configuration of a texturedirection map calculation unit according to the first embodiment.

FIG. 2B is a block diagram showing a detailed configuration of a texturedirection map calculation unit according to the first embodiment.

FIG. 3 is a block diagram showing a detailed configuration of ahole-filling processing unit according to the first embodiment.

FIG. 4 is a flowchart showing the operations of an apparatus forgenerating a new viewpoint image according to the first embodiment.

FIG. 5A shows an example of a depth map according to the firstembodiment.

FIG. 5B shows an example of a depth map according to the firstembodiment.

FIG. 5C shows an example of a depth map according to the firstembodiment.

FIG. 6 shows an example of a method for selecting a representativebackground pixel according to the first embodiment.

FIG. 7A shows an example of an input image according to the firstembodiment.

FIG. 7B shows an example of a generated new viewpoint image includingmissing pixels according to the first embodiment.

FIG. 8 is a flowchart showing a detailed procedure for calculating atexture direction map according to the first embodiment.

FIG. 9 is another flowchart showing a detailed procedure for calculatinga texture direction map according to the first embodiment.

FIG. 10 shows an example of calculation for the texture directions ofrepresentative background pixels selected in an input image according tothe first embodiment.

FIG. 11 is an illustration to explain the concept of a method forcalculating a texture direction map for missing pixels according to thefirst embodiment.

FIG. 12 is an illustration to explain the concept of a method forfilling a missing pixel region according to the first embodiment.

FIG. 13 is a flowchart showing an example of a detailed method forfilling missing pixels according to the first embodiment.

FIG. 14 shows an example of a new viewpoint image obtained bycomplementing the new viewpoint image including missing pixels that isshown in FIG. 7B.

FIG. 15 is a block diagram showing a configuration example of anapparatus for generating a new viewpoint image 200 according to amodification of the first embodiment.

FIG. 16A is an illustration to explain differences in advantage when auser adjusts or controls control parameters.

FIG. 16B is an illustration to explain differences in advantage when auser adjusts or controls control parameters.

FIG. 16C is an illustration to explain differences in advantage when auser adjusts or controls control parameters.

FIG. 17 is a block diagram showing a configuration of aplayback/recording device including a new viewpoint image generationmodule.

FIG. 18 is a block diagram showing a configuration of a displayincluding an apparatus for generating a new viewpoint image.

FIG. 19 is a block diagram showing a configuration of an imagingapparatus including an apparatus for generating a new viewpoint image.

DETAILED DESCRIPTION OF INVENTION Underlying Knowledge Forming Basis ofthe Present Disclosure

When a new viewpoint image is generated using an input image, a holeregion occurs in a region corresponding to an occluded region in aninput image (original viewpoint image). Therefore, it is necessary tofill a hole region using a hole-filling method or a disocclusion method.

As a well-known hole-filling method, an interpolation (or extrapolation)method based on a bilinear interpolation using pixels on the both sides(or the background side) of a hole region is known.

However, although working well for a small hole region, this method maycause problems such as blurred edges, boundary artifacts, and objectdistortions for a large hole region. When a new viewpoint image havingartifacts is used for a 3D image, 3D display quality significantlydecreases.

Moreover, techniques such as symmetric or asymmetric depth map smoothingare known as a technique to reduce the size of a hole region. However,this method has a problem that geometric distortion occurs in an object.In other words, using a 3D image generated by this method reduces theeffects of 3D display.

On the other hand, Patent Literature 1 suggests an in-painting methodbased on a depth value. This in-painting method uses background pixelsto reduce geometric distortion and improve the quality of filling a holeregion. However, this in-painting method tends to produce blurredtexture around a hole region, and thus a clear and natural new viewpointimage cannot be generated.

Moreover, Patent Literature 2 suggests a method for filling a holeregion based on the direction of image features. This method for fillingthe hole region estimates the direction of image features and fills apixel in the hole region using a pixel value calculated from a pixel setin the estimated direction. However, this method for filling the holeregion often cannot correctly estimate the direction or obtain asatisfactory result. For example, when foreground objects are on theboth sides of a hole region, this method cannot estimate the correctdirection. Moreover, another problem is that this method is ineffectivefor non-linear texture.

Here, one non-limiting and exemplary embodiment provides an apparatusand a method for generating a new viewpoint image that can generate aclearer and more natural new viewpoint image.

According to an exemplary embodiment disclosed herein, an apparatus forgenerating a new viewpoint image according to an aspect of the presentdisclosure, generates, from an input image, a new viewpoint image thatis an image of the input image viewed from a new viewpoint, the inputimage including a foreground image region that is an image region of anobject and a background image region other than the foreground imageregion. The apparatus includes: a generation unit that shifts theforeground image region in the input image, based on a depth mapcorresponding to the input image to generate a new viewpoint imageincluding a missing pixel region; a selection unit that selectsbackground pixels located a predetermined number of pixels away fromedge pixels, from among background pixels included in the backgroundimage region, as representative background pixels, the edge pixelsindicating a boundary between the foreground image region and thebackground image region in the input image; a texture direction mapcalculation unit that calculates a texture direction of each of theselected representative background pixels, the texture direction beingan edge direction of texture in the background image region for each ofthe representative background pixels, and to calculate a texturedirection map indicating texture directions, each corresponding to oneof missing pixels, based on locations of the representative backgroundpixels and locations of the missing pixels included in the missing pixelregion; and a hole-filling processing unit that fills the missing pixelregion with the background pixels in accordance with the texturedirection map.

This configuration can realize an apparatus for generating a newviewpoint image that can generate a clearer and more natural newviewpoint image.

More specifically, as the apparatus for generating a new viewpoint imageaccording to the present aspect can generate clear and natural imagetexture, it is possible to reduce blur and boundary artifacts whilemaintaining the shape of a foreground image region. This can improveimage quality, and thus better 3D effects can be obtained. In otherwords, it is possible to reduce discomfort when a user is viewing a 3Dimage.

Moreover, an apparatus for generating a new viewpoint image according tothe aspect can correctly calculate the direction of background texturein various cases. Therefore, the quality of the output image improves.

For example, the texture direction map calculation unit may include: atexture direction calculation unit that calculates texture directions ofthe representative background pixels; and a map calculation unit thatcalculates the texture direction map by weighting the texture directionsof the representative background pixels based on the locations of therepresentative background pixels and the locations of the missing pixelsand calculating the texture directions, each corresponding to one of themissing pixels.

For example, the map calculation unit may calculate the texturedirections, each corresponding to one of the missing pixels, by (i)calculating a direction in which weighted texture directions of therepresentative background pixels are summed, as one of preliminarytexture directions of the missing pixels, in accordance with distancesbetween one of the missing pixels and the representative backgroundpixels and directions of the representative background pixels withrespect to one of the missing pixels and (ii) averaging each of thecalculated preliminary texture directions of the missing pixels.

For example, the reliability value calculation unit may calculate areliability value for a texture direction of a representative backgroundpixel to be calculated, based on (i) a distance from a calculatedrepresentative background pixel to the representative background pixelto be calculated and (ii) an orientation of a vector of the calculatedrepresentative background pixel with respect to the representativebackground pixel to be calculated.

For example, the reliability value calculation unit may calculate areliability value for a texture direction of a representative backgroundpixel to be calculated, based on (i) a distance from a calculatedrepresentative background pixel to the representative background pixelto be calculated and (ii) an orientation of a vector of the calculatedrepresentative background pixel with respect to the representativebackground pixel to be calculated.

For example, the texture direction calculation unit may trace edges oftexture of the background image region from the selected representativebackground pixels as starting points to calculate the texture directionsof the selected representative background pixels.

For example, the apparatus for generating a new viewpoint imageaccording to an aspect of the present disclosure may further include anedge strength calculation unit that calculates edge strength for each ofthe selected representative background pixels, in which when the edgestrength of the representative background pixel is greater than athreshold, the texture direction map calculation unit may calculate atexture direction of the representative background pixel.

For example, the apparatus for generating a new viewpoint imageaccording to an aspect of the present disclosure may further include anedge detection unit that detects, based on the depth map, edge pixelsindicating a boundary between the foreground image region and thebackground image region in the input image, in which the selection unitmay select, as the representative background pixels, background pixelslocated a predetermined number of pixels away from the edge pixels in apredetermined direction, the edge pixels being detected by the edgedetection unit.

For example, the selection unit may, for each of the edge pixelsdetected by the edge detection unit, calculate an average value ofhorizontal differential coefficients and an average value of verticaldifferential coefficients for the edge pixel, determine an orthogonaldirection of the edge pixel as the predetermined direction, using theaverage value of the horizontal differential coefficients and theaverage value of the vertical differential coefficients, and select, asone of the representative background pixels, a background pixel locateda predetermined number of pixels away from the edge pixel in thedetermined orthogonal direction.

For example, the generation unit may generate a new viewpoint imageincluding the missing pixel region by (i) shifting the foreground imageregion in the input image, based on a depth value in the depth map, adistance between a right eye and a left eye, and a zero eye position and(ii) shifting the depth value in a region of the depth map correspondingto the foreground image region.

For example, the apparatus for generating a new viewpoint imageaccording to an aspect of the present disclosure may further include afilter that filters a new depth map corresponding to the new viewpointimage to smooth a depth value that is a depth value in a region of thenew depth map corresponding to the missing pixel region and is a depthvalue less than or equal to a predetermined value.

For example, the filling unit may include: a direction obtaining unitthat refers to the calculated texture direction map to obtain thetexture directions, each corresponding to one of the missing pixels; adistance determination unit that calculates distances from edge pixelsto the missing pixels, the edge pixels being in the obtained texturedirections, each corresponding to one of the missing pixels; a pixelobtaining unit that obtains, based on the distances, background pixelsin the corresponding texture directions; and a filling unit thatcalculates pixel values of missing pixels, using the obtained backgroundpixels to fill the missing pixel region, the pixel values being valueshaving the corresponding texture directions.

It should be noted that these general and specific aspects may beimplemented using a system, a method, an integrated circuit, a computerprogram or a computer-readable recording medium such as a CD-ROM, or anycombination of systems, methods, integrated circuits, computer programs,or recording media.

With reference to the drawings, the following specifically describes anapparatus and a method for generating a new viewpoint image according toan aspect of the present disclosure.

It should be noted that each of the exemplary embodiments describedbelow shows a general or specific example. The numerical values, shapes,materials, structural elements, the arrangement and connection of thestructural elements, steps, the processing order of the steps and so onshown in the following exemplary embodiments are mere examples, and arenot intended to limit the present disclosure. Therefore, among thestructural elements in the following exemplary embodiments, structuralelements not recited in any one of the independent claims representingsuperordinate concept are described as arbitrary structural elements.

Embodiment 1

FIG. 1 is a block diagram showing a configuration example of anapparatus for generating a new viewpoint image 100 according to thepresent embodiment.

The apparatus for generating a new viewpoint image 100 shown in FIG. 1generates, from an input image, a new viewpoint image that is an imageof the input image viewed from a new viewpoint. Specifically, theapparatus for generating a new viewpoint image 100 separately takes inan input image and a depth map, and generates a new viewpoint imagebased on these input image and depth map. The following explanationassumes that the input image includes a foreground image region that isthe image region of an object and a background image region other thanthe foreground image region, and the depth map includes depthinformation on the foreground and background image regions such asinformation indicating the distance between scene objects.

The apparatus for generating a new viewpoint image 100 includes a depthedge detection unit 102, a representative background pixel selectionunit 104, a new viewpoint image generation unit 106, a texture directionmap calculation unit 108, and a hole-filling processing unit 110.

The depth edge detection unit 102 detects, based on a depth mapcorresponding to an input image, edge pixels in the input image thatindicate a boundary between foreground and background image regions. Inother words, the depth edge detection unit 102 detects edge pixellocations in the depth map that indicate boundary points between aforeground object (foreground image region) and a background object(background image region). The depth edge detection unit 102 can thusdetect a boundary between the foreground and background image regions inthe input image.

The representative background pixel selection unit 104 is an example ofa selection unit, and selects, from background pixels included in abackground image region, background pixels located a predeterminednumber of pixels away from the edge pixels indicating the boundarybetween the foreground and background image regions in the input image.

Specifically, the representative background pixel selection unit 104selects, as representative background pixels, background pixels locateda predetermined number of pixels away from edge pixels in apredetermined direction. Here, the edge pixels are detected by the depthedge detection unit 102. In other words, the representative backgroundpixel selection unit 104 selects, as representative background pixels,background pixels located in a predetermined direction and distance fromthe locations of edge pixels in the input image corresponding to thedetected edge pixels in the depth map.

Here, for example, the representative background pixel selection unit104 may, for each of the edge pixels detected by the depth edgedetection unit 102, (i) calculate the average value of horizontaldifferential coefficients and the average value of vertical differentialcoefficients for the edge pixel, (ii) determine the orthogonal directionof the edge pixel, using the calculated average values of the horizontaland vertical differential coefficients, and (iii) select, as arepresentative background pixel, a background pixel located apredetermined number of pixels away from the edge pixel in thedetermined orthogonal direction.

The new viewpoint image generation unit 106 is an example of ageneration unit, and generates a new viewpoint image including a missingpixel region by shifting the foreground image region in the input imagebased on the depth map. In other words, the new viewpoint imagegeneration unit 106 generates a new viewpoint image including missingpixels, based on the input image and the depth map.

For instance, the new viewpoint image generation unit 106 generates anew viewpoint image including a missing pixel region by (i) shifting theforeground image region in the input image, based on a depth valueindicated by the depth map, the distance between right and left eyes,and zero eye position and (ii) shifting the depth value in a region of adepth map corresponding to the foreground image region.

Thus, in the present embodiment, the new viewpoint image generation unit106 not only generates a new viewpoint image by shifting the inputimage, but also generates, in the same way, a new depth mapcorresponding to a new viewpoint image by shifting the depth map. Itshould be noted that details will be described later.

The texture direction map calculation unit 108 (i) calculates thetexture directions of representative background pixels which are theedge directions of texture of the background region from the selectedrepresentative background pixels as starting points and (ii) calculate atexture direction map indicating texture directions, each correspondingto one of missing pixels, based on the locations of the representativebackground pixels and the locations of the missing pixels included inthe missing pixel region.

In other words, the texture direction map calculation unit 108calculates a texture direction map indicating texture directions in abackground image region (hereinafter referred to also as x-y directions)for missing pixels. It should be noted that the x-y directions are usedfor obtaining background pixels used for calculation of respective pixelvalues with which each missing pixel should be filled.

FIGS. 2A and 2B are block diagrams showing detailed configurations of atexture direction map calculation unit according to the presentembodiment. As shown in FIG. 2A, a texture direction map calculationunit 108A may include a texture direction calculation unit 1081 and amap calculation unit 1082. It should be noted that as shown in FIG. 2B,a texture direction map calculation unit 108B may further include areliability value calculation unit 1083 and an edge strength detectionunit 1084. The following is an example that the texture direction mapcalculation unit 108 has the configuration shown in FIG. 2B.

The reliability value calculation unit 1083 calculates a reliabilityvalue for the texture direction of each of the calculated representativebackground pixels. Specifically, the reliability value calculation unit1083 calculates a reliability value for the texture direction of arepresentative background pixel to be calculated, based on (i) thedistance from a calculated representative background pixel to therepresentative background pixel to be calculated and (ii) theorientation of a vector of the calculated representative backgroundpixel with respect to the representative background pixel to becalculated. It should be noted that the reliability value indicates theaccuracy of the calculated texture direction.

The edge strength detection unit 1084 calculates edge strength for eachrepresentative pixel selected by the representative background pixelselection unit 104.

The texture direction calculation unit 1081 calculates the texturedirections of the representative background pixels. Specifically, thetexture direction calculation unit 1081 calculates texture directions bytracing edges of texture in a background image region from the selectedrepresentative background pixels as starting points. Here, for example,when the edge strength of one of the representative background pixels isgreater than a threshold, the texture direction calculation unit 1081may calculate the texture direction of the representative backgroundpixel.

The map calculation unit 1082 weights the texture directions ofrepresentative background pixels, based on the locations of therepresentative background pixels and the locations of the missing pixelsto calculate texture directions, each corresponding to one of themissing pixels, and thus calculate a texture direction map indicatingthe texture directions, each corresponding to one of the missing pixels.

Here, for example, the map calculation unit 1082 may calculate (i) adirection in which weighted texture directions of representativebackground pixels are summed (blended), as one of the preliminarytexture directions of the missing pixels, in accordance with distancesbetween one of the missing pixels and the representative backgroundpixels and the directions of the representative background pixels withrespect to one of the missing pixels and (ii) texture directions, eachcorresponding to one of the missing pixels by averaging each of thecalculated preliminary texture directions of the missing pixels.Moreover, for example, the map calculation unit 1082 weights the texturedirection or texture directions of at least one first representativebackground pixel, based on the location or locations of the at least onefirst representative background pixel for which a reliability valuegreater than a predetermined threshold value is calculated by thereliability value calculation unit 1083, among the representativebackground pixels to calculate texture directions, each corresponding toone of the missing pixels.

The following description is based on FIG. 1. The hole-fillingprocessing unit 110 is an example of a hole-filling processing unit, andfills a missing pixel region, using background pixels obtained based ona texture direction map. In other words, the hole-filling processingunit 110 fills each missing pixel, using a background pixel obtained foreach missing pixel based on the calculated texture direction map. Thus,an output image (new viewpoint image) can be obtained.

FIG. 3 is a block diagram showing a detailed configuration of ahole-filling processing unit according to the present embodiment. Asshown in FIG. 3, a hole-filling processing unit 110A may include adirection obtaining unit 1101, a distance determining unit 1102, abackground pixel obtaining unit 1103, and a hole filling unit 1104.

The direction obtaining unit 1101 refers to a calculated texturedirection map to obtain texture directions, each corresponding to one ofthe missing pixels. The distance determining unit 1102 calculatesdistances from edge pixels to the missing pixels, the edge pixels beingin texture directions, each corresponding to one of missing pixelsobtained by the direction obtaining unit 1101. The background pixelobtaining unit 1103 obtains background pixels in the correspondingtexture directions, based on the distances calculated by the distancedetermining unit 1102. The hole filling unit 1104 is an example of afilling unit, and calculates pixel values for missing pixels, using thebackground pixels obtained by the background pixel obtaining unit 1103to fill a missing pixel region. Here, the pixel values have thecorresponding texture directions.

It should be noted that the apparatus for generating a new viewpointimage 100 configured as above and structural elements (units) includedtherein are typically achieved in the form of integrated circuit (ICs),application-specific integrated circuits (ASIC), large scale integrated(LSI) circuits, digital signal processor (DSP), or achieved by any CPUbased processor, such as ARM, and machine including personal computer(PC). Each of these modules can be in many single-function LSIs, or canbe in one integrated LSI. The name used here is LSI, but it may also becalled IC, system LSI, super LSI, or ultra LSI in accordance with thedegree of integration. Moreover, ways to achieve integration are notlimited to the LSI, and a special circuit or a general purpose processormay also achieve the integration. This includes a specializedmicroprocessor such as digital signal processor (DPS) that can bedirected by the program instruction. Field Programmable Gate Array(FPGA) that can be programmed after manufacturing LSI or areconfigurable processor that allows re-configuration of the connectionor configuration of LSI can be used for the same purpose. In the future,with advancement in manufacturing and process technology, a brand-newtechnology may replace LSI. The integration can be achieved by such atechnology. As an embodiment, the apparatus for generating a newviewpoint image 100 may be incorporated into imaging devices such asdigital still cameras and movie cameras. Moreover, as shown in a generalpurpose imaging system, the apparatus for generating a new viewpointimage 100 may be implemented in a standalone device to work with animaging system but other implementations are also possible. Theapparatus for generating a new viewpoint image 100 may be implemented inother types of devices.

The following describes the operations of the apparatus for generating anew viewpoint image 100 configured as above.

FIG. 4 is a flowchart showing the operations of an apparatus forgenerating a new viewpoint image according to the present embodiment.

First, the apparatus for generating a new viewpoint image 100 separatelytakes in an input image and a depth map.

The depth edge detection unit 102 detects edge pixels in the depth mapthat indicate boundary points between foreground and background imageregions (S302). The depth edge detection unit 102 can detect edge pixelsin the input image that indicate boundary points between the foregroundand background regions, based on the edge pixels in the depth map.

The representative background pixel selection unit 104 selects, asrepresentative background pixels, background pixels located in apredetermined direction and distance from the locations of the detectededge pixels in the input image (S304). For example, the representativebackground pixel selection unit 104 selects, as representativebackground pixels, background pixels located in a predetermineddirection and distance from the edge pixels in the input imagecorresponding to the detected edge pixels in the depth map.

The new viewpoint image generation unit 106 generates a new viewpointimage including missing pixels by shifting the foreground image regionof the input image based on the depth map (S306).

The texture direction map calculation unit 108 calculates a texturedirection map indicating the x-y directions of missing pixels (S308).

Finally, at step S310, missing pixels are filled using background pixelsobtained based on the calculated texture direction map (S310).

The apparatus for generating a new viewpoint image operates as above,and generates a new viewpoint image that is an image of an input imageviewed from a new viewpoint, from the input image and the correspondingdepth map.

FIGS. 5A to 5C show examples of a depth map according to the presentembodiment.

As shown in FIG. 5A, a depth map 400 has the depth values of aforeground object 410 and a background object 420. Here, the foregroundobject 410 and the background object 420 in the depth map 400 correspondto a foreground object (foreground image region) and a background object(background image region) in an input image, respectively.

Observing the depth map 400 reveals an occluded region and a hole region(hereinafter referred to also as a missing pixel region) in a newviewpoint image to be generated. This is because the foreground imageregion in an input image corresponding to the foreground object 410having a small depth value always occludes the background image regionin the input image corresponding to the background object 420 having alarge depth value. Here, in FIG. 5A, the foreground object 410 is abrighter color image region and the background object 420 is a darkercolor image region.

Therefore, a location where a hole (missing pixel) occurs can beidentified based on the difference between depth values. For example,this can be achieved by performing a simple differentiation for thehorizontal direction of the depth map 400. More specifically, a locationwhere the difference between depth values on the depth map 400 isgreater than a threshold (e.g., three) is detected as a depth edgepixel.

Here, when a right viewpoint image is generated from an input image,only the right depth edge pixels of the foreground object 410 may bedetected, for example, as shown in a depth map 402 in FIG. 5B. On theother hand, when a left viewpoint image is generated from the inputimage, only the left depth edge pixels of the foreground object 410 maybe detected, for example, as shown in a depth map 404 in FIG. 5C. Inother words, detected right depth edge pixels 7R and left depth edgepixels 7L indicate boundary points between the foreground object 410 andthe background object 420.

It should be noted that the sign (+, −) of a differentiation result maydifferentiate the right and left depth edge pixels of the foregroundobject 410.

Here, the right depth edge pixels of the depth map can be calculated,for example, as the following equation 1.

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 1} \right\rbrack & \; \\{{e_{D}(p)} = \left\{ \begin{matrix}{255,} & {{{if}\mspace{14mu}\left( {{D(p)} - {D\left( {p + 1} \right)}} \right)} > {thr}} \\{0,} & {otherwise}\end{matrix} \right.} & \left( {{Equation}\mspace{14mu} 1} \right)\end{matrix}$

Here, D(p) denotes an inputted depth map. p denotes the vector of (x,y)th location (pixel location) in the depth map. Moreover, p+1 denotesthe next pixel location in the x direction.

The following describes the details of S304 in FIG. 4 with reference toFIG. 6. FIG. 6 shows an example of a method for selecting arepresentative background pixel according to the present embodiment.

As described above, the representative background pixel selection unit104 selects, as representative background pixels, background pixelslocated in a predetermined direction and distance from edge pixels inthe input image corresponding to detected edge pixels in the depth map.The selected representative background pixels are used for calculationof a texture direction map. It should be noted that the representativebackground pixel selection unit 104 may select, as representativebackground pixels, background pixels in the input image corresponding tobackground pixels located in a predetermined direction and distance fromdetected edge pixels in the depth map.

For example, as shown in FIG. 5B or 5C, the representative backgroundpixel selection unit 104 can determine, from edge pixels in the depthmap 400 detected by the depth edge detection unit 102, boundary pointsbetween the foreground object 410 (corresponding to a foreground imageregion in the input image) and the background object 420 (correspondingto a background image region in the input image). Therefore, therepresentative background pixel selection unit 104 can selectrepresentative background pixels.

Here, the following describes an example of a region 510 including theboundary points between the foreground object 410 and the backgroundobject 420.

The representative background pixel selection unit 104 may determine adepth edge direction D501 that is an edge direction for an edge pixel520 in the region 510. Here, the representative background pixelselection unit 104 may determine the depth edge direction D501 by, forexample, calculating horizontal and vertical differential coefficientson the depth map for the pixels adjacent to the edge pixel 520.Moreover, the representative background pixel selection unit 104 maydetermine the depth edge direction D501 by calculating horizontal andvertical differential coefficients using, for example, the well-knownSobel operator.

The representative background pixel selection unit 104 determines adirection D503 corresponding to the predetermined direction. It shouldbe noted that the direction D503 is orthogonal with respect to the depthedge direction D501 (an angle 530 is 90 degrees). Of course, thedirection D503 does not have to be orthogonal with respect to the depthedge direction D501 (the angle 530 does not have to be 90 degrees).

The representative background pixel selection unit 104 determines thedistance of the direction D503 corresponding to the predetermineddistance, and selects, as representative background pixels, backgroundpixels located in this distance in the input image. It should be notedthat here, the locations of the selected background pixels on the depthmap 400 corresponds to the locations of representative background pixelsin the input image.

The following describes the details of S306 in FIG. 4. As mentionedabove, at S306, a new viewpoint image including missing pixels isgenerated by shifting a foreground image region in an input image basedon a depth map.

It should be noted that in the conventional DIBR method, a new viewpointimage is generated by directly shifting the input image to a targetviewpoint (sometimes called 3D warping). However, in the method, cracks(artifacts) in a new viewpoint image are often observed to occur. When a3D image is generated using a new viewpoint image having cracks(artifacts), the quality of the 3D image significantly decreases. Thesecracks (artifacts) are caused by depth quantization and rounding errorswhen a shift amount is calculated.

To avoid this problem, in the present embodiment, a depth map is shiftedin the same way as the input image is shifted. Thus, cracks in a newviewpoint image can be effectively removed from the shifted depth map(hereinafter referred to as a new depth map).

Here, a new depth map D_(N) can be calculated using Equation 2. InEquation 2, D denotes a depth map corresponding to the input image, andD_(N) denotes a shifted depth map, i.e., a new depth map.[Math. 2]D _(N)(p+d)=D(p)  (Equation 2)

It should be noted that in Equation 2, p denotes a vector indicating the(x, y)th location in the depth map, and d(=dx, 0) denotes a horizontalshifting vector.

Here, a shift amount d_(x) is expressed by Equation 3.[Math. 3]d _(x)=(Z−D(p))×B  (Equation 3)

In Equation 3, Z denotes a zero-parallax setting (zps) parameter, and Bdenotes a baseline parameter provided by a user. Here, the baselineparameter, for example, includes the distance between the right eye andleft eye of the user.

However, also in the present embodiment, cracks (artifacts) may occurbetween the foreground and background image regions of a new viewpointimage that is a shifted input image. The cracks occur inside theforeground image region and break the texture of the foreground imageregion in the new viewpoint image.

Therefore, in the present embodiment, to efficiently remove cracks, asimple 3×3 maximum filter may be applied to a new depth map (shifteddepth map) so that the texture of the foreground image region in the newviewpoint image is not broken.

In other words, the apparatus for generating a new viewpoint image 100may include a filter that filters a new depth map corresponding to a newviewpoint image to smooth a depth value that is a depth value in aregion of the new depth map corresponding to a missing pixel region andthat is a depth value less than or equal to a predetermined value.

It should be noted that not only the simple 3×3 maximum filter but alsoother types of filters may be used as a filter for removing cracks.

After the new depth map (shifted depth map) is generated, the newviewpoint image generation unit 106 generates a new viewpoint image,using a similar scheme.

For example, the new viewpoint image generation unit 106 can efficientlygenerate a new viewpoint image I_(N) from an input image I_(L), usingEquation 4 to correspond to a case when a camera is set parallel to theground and images for the left and right eyes are captured.[Math. 4]I _(N)(p)=I _(L)(p−d)  (Equation 4)

Here, d=(dx, 0) is a horizontal shifting vector, and d_(x) is expressedby Equation 5. It should be noted that z denotes a zero parallax setting(zps) parameter also in Equation 5. Moreover, B denotes a baselineparameter provided by a user, and D_(N) denotes a new depth map.[Math. 5]d _(x)=(Z−D _(N)(p))×B  (Equation 5)

With reference to FIGS. 7A and 7B, the following describes a case when anew viewpoint image including missing pixels is generated from an inputimage.

FIG. 7A shows an example of an input image according to the presentembodiment. FIG. 7B shows an example of a generated new viewpoint imageincluding missing pixels according to the present embodiment.

FIG. 7A shows an example of an input image 602 including a foregroundimage region 604 and a background image region 606. The depth map 400shown in FIG. 5A is an example of a depth map corresponding to the inputimage 602. The foreground image region 604 in the input image 602corresponds to the foreground object 410 in the depth map 400. Likewise,the background image region 606 in the input image 602 corresponds tothe background object 420 in the depth map 400.

FIG. 7B shows an example of a generated new viewpoint image 612including missing pixels and an example that the new viewpoint image 612is generated as an image for the right eye.

The new viewpoint image generation unit 106 generates the new viewpointimage 612 by shifting a foreground image region 614 in the input image602 based on the depth values and the parameters (B and Z) in the aboveequations. A hole region 618 (missing pixel region) occurs in the newviewpoint image 612. As shown in FIG. 7B, when an image for the righteye is generated, the hole region 618 appears on the right side of theforeground image region 614. However, it is necessary to fill the holeregion 618 to generate the new viewpoint image 612 as the best outputimage (image for the right eye).

It should be noted that in the present embodiment, since cracks areremoved from a new depth map corresponding to the new viewpoint image612, cracks (artifacts) are reduced in the new viewpoint image 612.

The details of S304 in FIG. 4 will be described with reference to FIG.8. FIG. 8 is a flowchart showing a detailed procedure for calculating atexture direction map according to the present embodiment.

As mentioned above, at S308, the texture direction map calculation unit108 calculates a texture direction map indicating the x-y directions ofmissing pixels (texture directions in a background image region).

At S304 shown in FIG. 4, representative background pixels are selectedin an input image. In this case, the edge strength detection unit 1084evaluates edge strength G for one of the representative backgroundpixels selected by the representative background pixel selection unit104 (S402).

The texture direction calculation unit 1081 compares the edge strength Gfor the representative background pixel and a threshold 1 (Thr1) (S404).At S404, when the edge strength G is greater than the threshold 1 (Thr1)(Yes at S404), the texture direction calculation unit 1081 proceeds toS406. It should be noted that when the edge strength G is less than orequal to the threshold 1 (Thr1) (No at S404), the texture directioncalculation unit 1081 goes back to S402. At S402, the edge strengthdetection unit 1084 evaluates edge strength G for a next one of therepresentative background pixels.

The texture direction calculation unit 1081 calculates the texturedirection of a representative background pixel for which the edgestrength is determined to be greater than the threshold (S406).

The reliability value calculation unit 1083 calculates the reliabilityvalue of the calculated texture direction (S408). Here, this reliabilityvalue indicates the accuracy of the calculated texture direction.

The map calculation unit 1082 compares the reliability value calculatedby the reliability value calculation unit 1083 and a threshold 2 (Thr2)(S410). At S410, when the reliability value is greater than thethreshold 2 (Thr2) (Yes at S410), the map calculation unit 1082 proceedsto S412. It should be noted that when the reliability value is less thanor equal to the threshold 2 (Thr2) (No at S10), the map calculation unit1082 goes back to S402. At S402, the edge strength detection unit 1084evaluates the edge strength G for a next one of the representativebackground pixels.

The map calculation unit 1082 weights the texture direction or texturedirections of at least one representative background pixel, based on thelocation or locations of the at least one representative backgroundpixel and the location of a missing pixel to calculate a texturedirection (x-y direction) corresponding to the missing pixel (S412).Here, the missing pixel is located in a predetermined distance from arepresentative background pixel. Moreover, since there are severalmissing pixels in the predetermined distance from the representativebackground pixel, x-y directions (preliminary texture directions), eachcorresponding to one of the missing pixels are calculated at S412. Itshould be noted that a x-y direction (a preliminary texture direction)calculated herein is a direction in which weighted directions of therepresentative background pixels are summed, in accordance with thedistances between one of the missing pixels and the representativebackground pixels and the directions of the representative backgroundpixels with respect to one of the missing pixels.

The map calculation unit 1082 checks whether all the representativebackground pixels selected by the representative background pixelselection unit 104 have been evaluated (S414). In other words, the mapcalculation unit 1082 checks whether all the missing pixels included ina missing pixel region have been evaluated, using the texture directionsof the representative background pixels selected by the representativebackground pixel selection unit 104.

At S414, when there is an unevaluated representative background pixelamong the representative background pixels selected by therepresentative background pixel selection unit 104 (No at S414), the mapcalculation unit 1082 goes back to S402. At S402, the edge strengthdetection unit 1084 evaluates the edge strength G for the nextrepresentative background pixel, i.e., one of the unevaluatedrepresentative background pixels.

It should be noted that when all the representative background pixelsselected by the representative background pixel selection unit 104 havebeen evaluated (Yes at S414), the map calculation unit 1082 proceeds tothe next step S416.

At S416, the map calculation unit 1082 averages each of the preliminarytexture directions of the calculated missing pixels to calculate texturedirections (resulting x-y directions), each corresponding to one of themissing pixels.

It should be noted that a detailed method for calculating a texturedirection map is not limited to the above case. Another example will bedescribed with reference to FIG. 9.

FIG. 9 is another flowchart showing a detailed procedure for calculatinga texture direction map according to the present embodiment. It shouldbe noted that since S502 to S506 correspond to S402 to S406 in FIG. 8and S510 to S516 correspond to S410 to S416 in FIG. 8, explanation forthese steps will be omitted here.

At S506, the texture direction calculation unit 1081 calculates thetexture directions of representative background pixels.

The reliability value computation unit 1083 traces the calculatedtexture direction along the edge direction of a representativebackground pixel for which edge strength is determined to be greaterthan a threshold at S504 (S507), to calculate the reliability value ofthe calculated texture direction (S508). For example, the reliabilityvalue calculation unit 1083 calculates a reliability value for thetexture direction of a representative background pixel to be calculated,based on (i) the distance between the representative background pixel inthe texture direction to be calculated and a calculated representativebackground pixel and (ii) the orientation of the vector of the texturedirection of the calculated representative background pixel with respectto the texture direction of the representative background pixel to becalculated. In other words, the reliability value calculation unit 1083calculates a reliability value, based on the vector orientation of acurrent texture direction calculated by tracing at S507 and the vectororientation of the previously calculated texture direction.

FIG. 10 shows a calculation example of the texture directions ofrepresentative background pixels selected in an input image according tothe present embodiment. FIG. 10 shows brightness values for a portion ofthe input image.

Here, for example, a pixel 903 is one of the representative backgroundpixels selected as above. In this case, the texture directioncalculation unit 1081 may calculate the texture direction of the pixel903 by, for example, performing the convolution over a block B913centered at the pixel 903. It should be noted that the 3×3 Sobeloperator can be, for example, used for the convolution. Thus, thetexture direction calculation unit 1081 can obtain the texture directionof the pixel 903, i.e., gradients (brightness gradients) in the xdirection (horizontal direction) and the y direction (verticaldirection) of the pixel 903.

Moreover, the texture direction calculation unit 1081 may calculate atexture direction by evaluating the edge strength of a representativebackground pixel as above. For example, when the edge strength isevaluated using the Sobel operator, as having big differences in thebrightness values in the x direction, the pixel 903 has a stronggradient for the x direction. This means that the pixel 903 has an edgedirection, the same direction as y direction (direction D920).

On the other hand, the texture direction calculation unit 1081 maycalculate the texture direction of a pixel 901, using a block B911. Inthis case, since the pixel 901 does not have big differences in thebrightness values in the x and y directions, the pixel 901 has a weakgradient both for the x and y directions. Therefore, the edge strengthof the pixel 901 evaluated by the edge strength detection unit 1084 issmaller than the threshold Thr1. Thus, further processing is notperformed for the pixel 901.

It should be noted that although the direction D920 is obtained for thepixel 903 as mentioned above, the reliability value calculation unit1083 may trace the calculated texture direction along the calculatededge direction in order to further verify the accuracy of the calculatedtexture direction (here, direction D920).

Moreover, the reliability value calculation unit 1083 may evaluate, witha block 915, a pixel 905 that is a representative background pixel to becalculated, based on the direction D920 of the pixel 903 (calculatedtexture direction of a representative background pixel). Likewise, anext pixel 907 (a pixel 907 in the block B911) may be evaluated based onthe direction D920 of the pixel 903.

It should be noted that as mentioned above, the texture direction may becalculated by tracing an edge direction.

The following describes the further details of a method for calculatingthe texture direction of a representative background pixel.

pb denotes the location of a background pixel (pixel location) selectednear the foreground image region in an input image. It should be notedthat this pixel location corresponds to the location of an effectivebackground pixel near a hole region in a generated new viewpoint image.The gradient of this pixel location can be calculated using Equation 6.[Math. 6]g _(x) =S _(y)

I _(L)(p _(b)), and g _(y) =−S _(x)

I _(L)(p _(b))  (Equation 6)

Here, gx denotes a horizontal edge direction, and gy denotes a verticaledge direction.[Math. 7]

denotes a convolution operator.

Moreover, S_(x) and S_(y) are modified Sobel operators and are expressedas, for example, the following equation 7.

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 8} \right\rbrack & \; \\{{S_{x} = \begin{bmatrix}{- 3} & 0 & 3 \\{- 10} & 0 & 10 \\{- 3} & 0 & 3\end{bmatrix}},{S_{y} = \begin{bmatrix}{- 3} & {- 10} & {- 3} \\0 & 0 & 0 \\3 & 10 & 3\end{bmatrix}}} & \left( {{Equation}\mspace{14mu} 7} \right)\end{matrix}$

The edge strength G indicating gradient strength can be calculated usingEquation 8.[Math. 9]G=√{square root over (g _(x) ² +g _(y) ²)}  (Equation 8)

Here, when the edge strength G is greater than the threshold Thr1, edgedirections tx and ty are traced using Equations 9 to 11.[Math. 10]t _(x) ^((i)) =t _(x) ^((i-1)) +g _(x) ^((i)) , t _(y) ^((i)) =t _(y)^((i-1)) +g _(y) ^((i))  (Equation 9)and[Math. 11]g _(x) ^((i)) =S _(y)

I _(L)(p _(b)+α^((i-1)) t′ _(x) ^((i-1)))  (Equation 10)g _(y) ^((i)) =−S _(x)

I _(L)(p _(b)+α^((i-1)) t′ _(y) ^((i-1)))  (Equation 11)

Here, t′x and t′y are the normalized tx and ty, respectively, α is apredetermined operation step. i denotes the number of steps of tracingalong an edge direction. For example, eight steps may be used to ensurethat the edge (texture direction) is long enough. The texture directionmap of a missing pixel p is calculated using Equations 12 and 13.

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 13} \right\rbrack & \; \\{{\theta_{x}(p)} = {\frac{1}{\sum\limits_{k \in R}^{\;}\;{w_{k}(p)}}{\sum\limits_{k \in R}\;{{w_{k}(p)}{t_{x,k}^{\prime}(p)}}}}} & \left( {{Equation}\mspace{14mu} 12} \right) \\\left\lbrack {{Math}.\mspace{14mu} 14} \right\rbrack & \; \\{{\theta_{y}(p)} = {\frac{1}{\sum\limits_{k \in R}^{\;}\;{w_{k}(p)}}{\sum\limits_{k \in R}\;{{w_{k}(p)}{t_{y,k}^{\prime}(p)}}}}} & \left( {{Equation}\mspace{14mu} 13} \right)\end{matrix}$

It should be noted that when the direction of a pixel j is evaluated inthe above equation,[Math. 15]w _(j)(q)=l _(j) ²(q)/(h _(j) ²(q)+β)is a weighted value for a missing pixel location q.

lj is the distance between the pixel j and the pixel location q that isorthogonal with respect to a depth edge pixel. hj is the distancebetween a pixel q and a calculated direction. β is a constant. θ_(x) andθ_(y) are used in the hole filling steps.

FIG. 11 is an illustration to explain the concept of a method forcalculating a texture direction map for missing pixels according to thepresent embodiment.

FIG. 11 shows a new viewpoint image 1200 including a background imageregion 1201 and a foreground image region 1202. A missing pixel region1204 occurs in the new viewpoint image 1200. FIG. 11 also showsrepresentative background pixels e1 and e2, and missing pixels p1, p2,and p3, together with edge pixels 1240 and 1250.

A texture 1206 schematically shows the texture for the representativebackground pixel e1. The edge pixels 1240 and 1250 correspond to edgepixels that indicate the boundary between the foreground and backgroundimage regions 1202 and 1201 and that are detected in the new viewpointimage 1200. The representative background pixel e1 corresponds to arepresentative background pixel that is selected for the edge pixel1240. The representative background pixel e2 corresponds to arepresentative background pixel that is selected for the edge pixel1250.

A direction D1208 is a texture direction calculated for therepresentative background pixel e1. Here, a radius R1222 is determinedbased on the direction D1208, and missing pixels included in a circle1220 defined by the radius R1222 may be determined.

The following describes, as an example, missing pixels that are pixelsin the circle 1200 centered at the representative background pixel e1.

The x-y direction (texture direction) of the missing pixel p1 iscalculated by weighting the texture direction of the representativebackground pixel e1. This weight (weighted value) is, for example,calculated based on the distance between the missing pixel p1 and therepresentative background pixel e1 (first distance) and the distancefrom the line in the direction D1208 to the missing pixel p1 (seconddistance). In the example shown in FIG. 11, as the missing pixel p1 isin the line in the direction D1208, the second distance is zero.Therefore, the texture direction of the missing pixel p1 is calculatedbased on the direction D1208 and the first distance.

Likewise, the x-y direction (texture direction) of the missing pixel p2is calculated by weighting the texture direction of the representativebackground pixel e1. In other words, the x-y direction (texturedirection) of the missing pixel p2 is calculated based on the directionD1208 and both first and second distances. Here, the first distance isthe distance between the missing pixel p2 and the representativebackground pixel e1. The second distance is the distance from the linein the direction D1208 to the missing pixel p2. In the example shown inFIG. 11, the weighted value of the missing pixel p1 is greater than theweighted value of the missing pixel p2.

Moreover, the missing pixel p3 is located both in the circles 1220 and1230. In this case, as shown in Equations 10 and 11, the texturedirection (direction D1208) of the representative background pixel e1and the texture direction of the representative background pixel e2 areweighted (blended) to calculate the texture direction of the missingpixel p3.

Thus, after the texture directions of all the missing pixels (texturedirection map) are calculated, it is possible to fill the missing pixelregion 1204 (hole region).

It should be noted that since information on the hole region (missingpixel region) cannot be obtained from an input image, filling a holeregion is not a simple task. The important clue is that a hole region isa background image region in a new viewpoint image.

Therefore, in the present embodiment, a hole region is filled using thetexture directions of representative background pixels. In other words,it is possible to appropriately fill the hole region, using backgroundpixels (texture pixels) in the background image region that are obtainedusing texture direction maps θ_(x) and θ_(y).

This will be described using the following equations. The missing pixelp in a generated new viewpoint image can be calculated using Equations14 and 15.[Math. 16]I _(N)(p)=I _(N)(p+Δ)  (Equation 14)[Math. 17]Δ=(2λθ′_(x)(p),2λθ′_(y)(p))  (Equation 15)

In the above equations, θ′_(x)(p) and θ′_(y)(p) denote the normalizeddirections of the missing pixel p. λ denotes the distance between themissing pixel p and a non-missing pixel (background pixel) closest tothe missing pixel p in the calculated direction.

Here, to ensure that[Math. 18]I _(N)(p+Δ)is a background pixel, the corresponding location in the shifted depthmap (new depth map) is checked.

With reference to FIG. 12, a method for filling a missing pixel regionwill be described that uses background pixels obtained using a texturedirection map. FIG. 12 is an illustration to explain the concept of amethod for filling a missing pixel region according to the presentembodiment.

As shown in FIG. 12, a background pixel 1312 is extracted (flipped)along a texture direction D1304 calculated based on the direction D1208to fill a missing pixel 1302. Here, as mentioned above, λ denotes thedistance between the missing pixel 1302 and a background pixel closestto the missing pixel 1302 in the calculated texture direction D1304. Thedistance between the missing pixels 1302 and 1312 is 2λ.

Thus, a missing pixel region can be naturally filled with appropriatebackground pixels by using texture directions, each corresponding to oneof missing pixel locations for filling the missing pixel region.

It should be noted that to obtain smooth texture, missing pixels may befilled using bilinear interpolation in addition to the above method.Thus, a higher quality new viewpoint image can be obtained by fillingmissing pixels with more appropriate background pixels.

The following describes a detailed method for filling a missing pixelregion with reference to FIG. 13.

FIG. 13 is a flowchart showing an example of the detailed method forfilling missing pixels according to the present embodiment.

The hole-filling processing unit 110 obtains the x-y direction (texturedirection) of a missing pixel determined at S602 such as θ′_(x)(p) andθ′_(y)(p) of the missing pixel (S604). The hole-filling processing unit110 determines the distance (λ) from the missing pixel to a first validpixel (background pixel) in the obtained x-y direction (S606). Thehole-filling processing unit 110 obtains a background pixel based on thedetermined distance (λ) and the x-y direction (S608). The hole-fillingprocessing unit 110 calculates the pixel value of the missing pixel,based on the obtained background pixel (S610). The hole-fillingprocessing unit 110 determines whether or not all the missing pixelshave been filled (S610). At S612, when any missing pixels are determinedto be unfilled (No at S612), the hole-filling processing unit 110determines a next missing pixel and repeats the processing starting fromS604. On the other hand, when all the missing pixels are determined tobe filled (Yes at S612), the hole-filling processing unit 110 ends theprocessing.

In this way, the apparatus for generating a new viewpoint image 100 cangenerate a new viewpoint image (new view). For example, as shown in FIG.14, missing pixels are naturally filled with background pixels in aregion 618A. Here, FIG. 14 shows an example of the new viewpoint image612 obtained by complementing a new viewpoint image including missingpixels that is shown in FIG. 7B. Same reference numerals are given tothe same elements as those shown in FIG. 7B, and detailed descriptionwill be omitted here.

The present embodiment can achieve an apparatus and a method forgenerating a new viewpoint image that can generate a clearer and morenatural new viewpoint image.

For example, an apparatus and a method for generating a new viewpointimage according to the present embodiment calculates the orientation ofthe texture by tracing a strong texture gradient in a representativebackground pixel along the direction of the background pixel. Thetexture direction map of the missing pixels (pixels in a hole region) iscalculated based on the orientations of texture. Thus, missing pixelscan be filled using background pixels along the directions guided by thetexture direction map. Thus, the background texture directions can becorrectly calculated in various cases. This allows the quality of outputimages to improve.

It should be noted that an apparatus and a method for generating a newviewpoint image according to the present embodiment may, for example,(i) evaluate edge strength for each of the selected representativebackground pixels, (ii) calculate the texture direction of a selectedrepresentative background pixel for which the edge strength is greaterthan a threshold, and (iii) further calculate a reliability value bytracing the texture direction to calculate the orientation of thetexture. In these cases, background pixels for filling missing pixelsare obtained using the above information. In this way, both linear andnon-linear missing pixels (pixels in a hole region) can be naturallyfilled.

Thus, an apparatus and a method for generating a new viewpoint imageaccording to the present embodiment can produce natural texture in ahole region while reducing blur and preventing distortion in aforeground image region. In other words, as an apparatus and a methodfor generating a new viewpoint image according to the present embodimentcan produce clear and natural image texture, it is possible to reduceblur and boundary artifacts while maintaining the shape of a foregroundimage region.

Therefore, an apparatus and a method for generating a new viewpointimage according to the present embodiment can generate a high qualitynew viewpoint image even when a long baseline is synthesized. Moreover,using this new viewpoint image for 3D display can improve 3D quality. Inother words, an apparatus and a method for generating a new viewpointimage according to the present embodiment can be used for 3Dapplication.

Modification

It should be noted that in the above embodiment, a baseline parameterand a zero parallax parameter may be set according to input from a user.The following describes the details.

FIG. 15 is a block diagram showing a configuration example of anapparatus for generating a new viewpoint image 200 according to amodification of the present embodiment. Same reference numerals aregiven to the same elements as those shown in FIG. 1, and detaileddescription will be omitted here.

A new viewpoint image generation unit 206 in the apparatus forgenerating a new viewpoint image 200 shown in FIG. 15 is different fromthe new viewpoint image generation unit 106 in the apparatus forgenerating a new viewpoint image 100 shown in FIG. 1.

Specifically, the new viewpoint image generation unit 206 also takes ina control parameter signal S220 from a user to generate a new viewpointimage according to input from the user such as a baseline parameter (B)and a zero parallax setting parameter (z). As being the same asexplanation for the new viewpoint image generation unit 106, furtherdescription will be omitted here.

FIGS. 16A to 16C are illustrations to explain differences in advantagewhen a user adjusts or controls a control parameter. FIGS. 16A to 16Cshow a left eye EL and a right eye ER of the user, a foreground imageregion for the left eye 9L, a foreground image region 9R for the righteye, and an object PO displayed on a display screen 1300. Here, theforeground image region 9L for the left eye and the foreground imageregion for the right eye 9R correspond to foreground image regions in aninput image and a new viewpoint image generated from the input image.

FIG. 16A shows an example that parallax is positive and the object POappears deep-in from (i.e., behind) the display screen 1300. FIG. 16Bshows an example that parallax is zero and the object PO appears on thedisplay screen 1300. Moreover, FIG. 16C shows an example that parallaxis negative and the object PO appears to pop out from (i.e., in frontof) the display screen 1300.

Thus, the user can control the object to have more or less difference inhow the object appears with respect to the display screen by adjustingthe baseline parameter (B). Moreover, the user can control the object toobtain a pop-out effect and a deep-in effect by adjusting the zeroparallax setting parameter (z).

As mentioned above, an apparatus and a method for generating a newviewpoint according to the present embodiment can generate a shifteddepth map by shifting an inputted depth map based on the value of theinputted depth map and input from the user.

It should be noted that as same as the present embodiment, a filter maybe applied to the new depth map in order to remove small holes (cracks)from the shifted depth map (new depth map).

In this case, a new viewpoint image is generated based on the inputimage and the new depth map after filtering. As an advantage for this,crack artifacts are completely removed from the foreground image regionof the generated image. Therefore, the quality of an output imageincreases.

Embodiment 2

The present embodiment describes an example of an apparatus including anapparatus for generating a new viewpoint image described in the firstembodiment.

FIG. 17 is a block diagram showing a configuration of aplayback/recording device 1400 including a new viewpoint imagegeneration module 1408. The device 1400 takes in image/video data usinga storage media reader 1402, and sends a signal S22 to an image decoder1404. When the image/video data is encoded, the image decoder decodesthe image/video data and sends the decoded image/video data to a depthmap generation module 1406. The depth map generation module 1406generates a depth map and sends the generated depth map to a newviewpoint image generation module 1408. The new viewpoint imagegeneration module 1408 generates a new viewpoint image and sends thegenerated new viewpoint image to an image output module 1410. The imageoutput module 1410 outputs an image that includes a generated newviewpoint image and that can be used for display at a display unit 1414,for storage/transmission, or for printing. Here, the new viewpoint imagegeneration module 1408 corresponds to the apparatus for generating a newviewpoint image described in the first embodiment.

FIG. 18 is a block diagram showing a configuration of a display 1500including an apparatus for generating a new viewpoint image. A 3D imageprocessor 1510 may provide an apparatus for generating a new viewpointimage in the processor with an input image and a depth map. 3D image maybe displayed on the display unit of the display 1500. A user may controlparameters for the apparatus for generating a new viewpoint image, usinga remote control or other similar devices, to control 3D effects.

The 3D image processor 1510 and other modules therein are typicallyachieved in the form of integrated circuits (ICs), application-specificintegrated circuits (ASIC), large scale integrated (LSI) circuits, ordigital signal processor (DSP). Each of these modules can be in manysingle-function LSIs, or also can be in one integrated LSI. The nameused here is LSI, but it may also be called IC, system LSI, super LSI,or ultra LSI in accordance with the degree of integration. Moreover,ways to achieve integration are not limited to the LSI, and a specialcircuit or a general purpose processor may also achieve the integration.This includes a specialized microprocessor such as digital signalprocessor (DSP) that can be directed by the program instruction. FieldProgrammable Gate Array (FPGA) that can be programmed aftermanufacturing LSI or a reconfigurable processor that allowsre-configuration of the connection or configuration of LSI can be usedfor the same purpose. In the future, with advancement in manufacturingand process technology, a brand-new technology may replace LSI. Theintegration can be achieved by such a technology.

FIG. 19 is a block diagram showing a configuration of an imagingapparatus 1600 including an apparatus for generating a new viewpointimage. The imaging apparatus 1600 includes an optical system 1602, animage sensor(s) 1604, an analog-to-digital converter (ADC) 1606, animage processor 1608, a microcomputer 1610, an external memory 1612, adriver controller 1620, an optical image stabilizer (OIS) sensor 1618,an operation unit 1622, a storage/transmission device 1616, and adisplay device 1614.

The image processor 1608 includes an internal memory 1640, a raw imageprocessor 1642, a color image processor 1643, and a 3D image processor1644. The 3D image processor 1644 includes the apparatus for generatinga new viewpoint image according to the first embodiment. On the otherhand, other components such as a microphone and a speaker are not shown.However, this does not limit the scope and spirit of the presentdisclosure.

The optical system 1602 may include components such as lenses or a setof lenses, zoom/focus mechanisms, actuators, shutters, and apertures inorder to control light signals reaching the image sensor(s) 1604. Theimage sensor 1604 accumulates incoming light signals and converts thelight signals into electrical signals. Moreover, the microcomputer 1610directs the image sensor 1604. The electrical signals are converted intodigital data (raw image data) by the ADC 1606 and stored in the internalmemory 1640 or the external memory 1612. The raw image processor 1642may take in the raw image data from the internal memory 1640 (or theexternal memory 1612) and perform many pre-processing (not shown in FIG.19) such as resizing, linearity correction, white balance, and gammacorrection. This pre-processed raw image can be stored or transmitted bythe storage/transmission device 1616. The pre-processed raw image canalso be processed by the color image processor 1643 to generate a colorimage such as RGB (three primary colors of light) or YCbCr. To generatea desired color image, the color image processor 1643 may perform colorinterpolation, color correction, tonal rage adjustment, color noisereduction, and other processing. The 3D image processor 1644 may take inone or more images (such as stereo images) and may re-generate 3D imagesusing an apparatus for generating a new viewpoint image included in the3D image processor 1644. When displayed on the display device 1614, theoutput 3D image(s) may be used for visualization and may also be storedin the storage/transmission device 1616 for further use. Storagedevices, for example, include but are not limited to flash-based memorycards, hard drivers, and optical devices. Transmission devices, forexample, include but are not limited to a HDMI interface, a USBinterface, a wireless interface and a direct-to-printer interface. Thestorage or transmission device may optionally consist of lossless orlossy compression.

The optical system 1602 may be controlled by the driver controller 1620which is directed by the microcomputer 1610. The operation portion 1622receives user operation input and sends electrical signals to themicrocomputer 1610. The microcomputer 1610 then directs related modulescorresponding to the user input, such as the driver controller 1620, theimage sensor(s) 1604, and the image processor 1608. The OIS sensor 1618detects motion due to hand tremor or camera motion, and sends electricalsignals to the microcomputer 1610. To move the lenses compensating forthe motion, the microcomputer 1610 directs the driver controller 1620 tocontrol actuators or the like in the optical system 1202. This reducesblur caused by hand tremor or camera motion.

The image processor 1608, the 3D image processor 1644, and the modulesin the processors are typically achieved in the form of IntegratedCircuits (ICs), Application-Specific Integrated Circuits (ASICs), orLarge Scale Integrated (LSI) circuits. Each of these modules can be inmany single-function LSIs, or can be in one integrated LSI. The nameused here is LSI, but it may also be called IC, system LSI, super LSI,or ultra LSI in accordance with the degree of integration. Moreover,ways to achieve integration are not limited to the LSI, and a specialcircuit or a general purpose processor may also achieve the integration.This includes a specialized microprocessor such as DSP (digital signalprocessor) that can be directed by the program instruction. FieldProgrammable Gate Array (FPGA) that can be programmed aftermanufacturing LSI or a reconfigurable processor that allowsre-configuration of the connection or configuration of LSI can be usedfor the same purpose. In the future, with advancement in manufacturingand process technology, a brand-new technology may replace LSI. Theintegration can be achieved by such a technology.

It should be noted that each of the structural elements in each of theabove-described embodiments may be configured in the form of anexclusive hardware product, or may be realized by executing a softwareprogram suitable for the structural element. Each of the structuralelements may be realized by means of a program executing unit, such as aCPU and a processor, reading and executing the software program recordedon a recording medium such as a hard disk or a semiconductor memory.Here, the software program for realizing an image decoding device andothers according to each of the embodiments is a program describedbelow.

In other words, this program causes a computer to execute: a method forgenerating a new viewpoint image, which generates, from an input image,a new viewpoint image that is an image of the input image viewed from anew viewpoint, the input image including a foreground image region thatis an image region of an object and a background image region other thanthe foreground image region, the program including: shifting theforeground image region in the input image, based on a depth mapcorresponding to the input image to generate a new viewpoint imageincluding a missing pixel region; selecting background pixels located apredetermined number of pixels away from edge pixels, from amongbackground pixels included in the background image region, asrepresentative background pixels, the edge pixels indicating a boundarybetween the foreground image region and the background image region inthe input image; calculating a texture direction of each of the selectedrepresentative background pixels, the texture direction being an edgedirection of texture in the background image region for each of therepresentative background pixels, and to calculate a texture directionmap indicating texture directions, each corresponding to one of missingpixels, based on locations of the representative background pixels andlocations of the missing pixels included in the missing pixel region;and filling the missing pixel region with the background pixels inaccordance with the texture direction map.

Although an apparatus and a method for generating a new viewpoint imageaccording to only some exemplary embodiments of the present disclosurehave been described in detail above, those skilled in the art willreadily appreciate that many modifications are possible in the exemplaryembodiments without materially departing from the novel teachings andadvantages of the present disclosure. Accordingly, all suchmodifications are intended to be included within the scope of thepresent disclosure.

The present disclosure can be used for, for example, an apparatus and amethod for generating a new viewpoint image, and especially for, forexample, an apparatus and a method for generating a new viewpoint imagethat are included in imaging devices such as digital still cameras andmovie cameras and that can generate a new viewpoint image constituting astereoscopic image or a multi-view image for 3D display.

The invention claimed is:
 1. An apparatus for generating a new viewpointimage, which generates, from an input image, a new viewpoint image thatis an image of the input image viewed from a new viewpoint, the inputimage including a foreground image region that is an image region of anobject and a background image region other than the foreground imageregion, the apparatus comprising: a generation unit configured to shiftthe foreground image region in the input image, based on a depth mapcorresponding to the input image to generate a new viewpoint imageincluding a missing pixel region, according to a depth image basedrendering (DIBR) technique; a selection unit configured to selectbackground pixels located a predetermined number of pixels away fromedge pixels, from among background pixels included in the backgroundimage region, as representative background pixels, the edge pixelsindicating a boundary between the foreground image region and thebackground image region in the input image; a texture direction mapcalculation unit configured to calculate a texture direction of each ofthe selected representative background pixels, the texture directionbeing an edge direction of texture in the background image region foreach of the representative background pixels, and to calculate, for allof missing pixels, a texture direction map indicating texture directionseach corresponding to one of the missing pixels, based on locations ofthe representative background pixels and locations of the missing pixelsincluded in the missing pixel region; and a hole-filling processing unitconfigured to entirely fill, at one time, the missing pixel region withthe background pixels without using the pixels of the foreground imageregion, while considering continuity at a boundary with the backgroundimage region and without considering discontinuity at a boundary withthe foreground image region, in accordance with the texture directionmap indicating the texture directions of the missing pixels.
 2. Theapparatus for generating a new viewpoint image according to claim 1,wherein the texture direction map calculation unit includes: a texturedirection calculation unit configured to calculate texture directions ofthe representative background pixels; and a map calculation unitconfigured to calculate the texture direction map by weighting thetexture directions of the representative background pixels based on thelocations of the representative background pixels and the locations ofthe missing pixels and calculating the texture directions, eachcorresponding to one of the missing pixels.
 3. The apparatus forgenerating a new viewpoint image according to claim 2, wherein the mapcalculation unit is configured to calculate the texture directions, eachcorresponding to one of the missing pixels, by (i) calculating adirection in which weighted texture directions of the representativebackground pixels are summed, as one of preliminary texture directionsof the missing pixels, in accordance with distances between one of themissing pixels and the representative background pixels and directionsof the representative background pixels with respect to one of themissing pixels and (ii) averaging each of the calculated preliminarytexture directions of the missing pixels.
 4. The apparatus forgenerating a new viewpoint image according to claim 2, furthercomprising a reliability value calculation unit configured to calculatereliability values for the calculated texture directions of therepresentative background pixels, wherein the map calculation unit isconfigured to calculate the texture directions, each corresponding toone of the missing pixels, by weighting a texture direction of at leastone first representative background pixel, based on a location of the atleast one first representative background pixel and locations of themissing pixels, each of the first representative background pixels beinga pixel for which a reliability value greater than a predeterminedthreshold is calculated by the reliability value calculation unit, amongthe representative background pixels.
 5. The apparatus for generating anew viewpoint image according to claim 4, wherein the reliability valuecalculation unit is configured to calculate a reliability value for atexture direction of a representative background pixel to be calculated,based on (i) a distance from a calculated representative backgroundpixel to the representative background pixel to be calculated and (ii)an orientation of a vector of the calculated representative backgroundpixel with respect to the representative background pixel to becalculated.
 6. The apparatus for generating a new viewpoint imageaccording to claim 2, wherein the texture direction calculation unit isconfigured to trace edges of texture of the background image region fromthe selected representative background pixels as starting points tocalculate the texture directions of the selected representativebackground pixels.
 7. The apparatus for generating a new viewpoint imageaccording to claim 1, further comprising an edge strength calculationunit configured to calculate edge strength for each of the selectedrepresentative background pixels, wherein when the edge strength of therepresentative background pixel is greater than a threshold, the texturedirection map calculation unit is configured to calculate a texturedirection of the representative background pixel.
 8. The apparatus forgenerating a new viewpoint image according to claim 1, furthercomprising an edge detection unit configured to detect, based on thedepth map, edge pixels indicating a boundary between the foregroundimage region and the background image region in the input image, whereinthe selection unit is configured to select, as the representativebackground pixels, background pixels located a predetermined number ofpixels away from the edge pixels in a predetermined direction, the edgepixels being detected by the edge detection unit.
 9. The apparatus forgenerating a new viewpoint image according to claim 8, wherein theselection unit is configured to, for each of the edge pixels detected bythe edge detection unit, calculate an average value of horizontaldifferential coefficients and an average value of vertical differentialcoefficients for the edge pixel, determine an orthogonal direction ofthe edge pixel as the predetermined direction, using the average valueof the horizontal differential coefficients and the average value of thevertical differential coefficients, and select, as one of therepresentative background pixels, a background pixel located apredetermined number of pixels away from the edge pixel in thedetermined orthogonal direction.
 10. The apparatus for generating a newviewpoint image according to claim 1, wherein the generation unit isconfigured to generate a new viewpoint image including the missing pixelregion by (i) shifting the foreground image region in the input image,based on a depth value in the depth map, a distance between a right eyeand a left eye, and a zero eye position and (ii) shifting the depthvalue in a region of the depth map corresponding to the foreground imageregion.
 11. The apparatus for generating a new viewpoint image accordingto claim 1, further comprising a filter that filters a new depth mapcorresponding to the new viewpoint image to smooth a depth value that isa depth value in a region of the new depth map corresponding to themissing pixel region and is a depth value less than or equal to apredetermined value.
 12. The apparatus for generating a new viewpointimage according to claim 1, wherein the filling unit includes: adirection obtaining unit configured to refer to the calculated texturedirection map to obtain the texture directions, each corresponding toone of the missing pixels; a distance determination unit configured tocalculate distances from edge pixels to the missing pixels, the edgepixels being in the obtained texture directions, each corresponding toone of the missing pixels; a pixel obtaining unit configured to obtain,based on the distances, background pixels in the corresponding texturedirections; and a filling unit configured to calculate pixel values ofmissing pixels, using the obtained background pixels to fill the missingpixel region, the pixel values being values having the correspondingtexture directions.
 13. A method for generating a new viewpoint image,which generates, from an input image, a new viewpoint image that is animage of the input image viewed from a new viewpoint, the input imageincluding a foreground image region that is an image region of an objectand a background image region other than the foreground image region,the method comprising: shifting the foreground image region in the inputimage, based on a depth map corresponding to the input image to generatea new viewpoint image including a missing pixel region, according to adepth image based rendering (DIBR) technique; selecting backgroundpixels located a predetermined number of pixels away from edge pixels,from among background pixels included in the background image region, asrepresentative background pixels, the edge pixels indicating a boundarybetween the foreground image region and the background image region inthe input image; calculating a texture direction of each of the selectedrepresentative background pixels, the texture direction being an edgedirection of texture in the background image region for each of therepresentative background pixels, and to calculate, for all of missingpixels, a texture direction map indicating texture directions eachcorresponding to one of the missing pixels, based on locations of therepresentative background pixels and locations of the missing pixelsincluded in the missing pixel region; and filling entirely, at one time,the missing pixel region with the background pixels without using thepixels of the foreground image region, while considering continuity at aboundary with the background image region and without consideringdiscontinuity at a boundary with the foreground image region, inaccordance with the texture direction map indicating the texturedirections of the missing pixels.
 14. A non-transitory computer-readablerecording medium for use in a computer, the recording medium having acomputer program recorded thereon for causing the computer to execute: amethod for generating a new viewpoint image, which generates, from aninput image, a new viewpoint image that is an image of the input imageviewed from a new viewpoint, the input image including a foregroundimage region that is an image region of an object and a background imageregion other than the foreground image region, the program comprising:shifting the foreground image region in the input image, based on adepth map corresponding to the input image to generate a new viewpointimage including a missing pixel region, according to a depth image basedrendering (DIBR) technique; selecting background pixels located apredetermined number of pixels away from edge pixels, from amongbackground pixels included in the background image region, asrepresentative background pixels, the edge pixels indicating a boundarybetween the foreground image region and the background image region inthe input image; calculating a texture direction of each of the selectedrepresentative background pixels, the texture direction being an edgedirection of texture in the background image region for each of therepresentative background pixels, and to calculate, for all of missingpixels, a texture direction map indicating texture directions eachcorresponding to one of the missing pixels, based on locations of therepresentative background pixels and locations of the missing pixelsincluded in the missing pixel region; and filling entirely, at one time,the missing pixel region with the background pixels without using thepixels of the foreground image region, while considering continuity at aboundary with the background image region and without consideringdiscontinuity at a boundary with the foreground image region, inaccordance with the texture direction map indicating the texturedirections of the missing pixels.
 15. An integrated circuit forgenerating a new viewpoint image, which generates, from an input image,a new viewpoint image that is an image of the input image viewed from anew viewpoint, the input image including a foreground image region thatis an image region of an object and a background image region other thanthe foreground image region, the integrated circuit comprising: ageneration unit configured to shift the foreground image region in theinput image, based on a depth map corresponding to the input image togenerate a new viewpoint image including a missing pixel region,according to a depth image based rendering (DIBR) technique; a selectionunit configured to select background pixels located a predeterminednumber of pixels away from edge pixels, from among background pixelsincluded in the background image region, as representative backgroundpixels, the edge pixels indicating a boundary between the foregroundimage region and the background image region in the input image; atexture direction map calculation unit configured to calculate a texturedirection of each of the selected representative background pixels, thetexture direction being an edge direction of texture in the backgroundimage region for each of the representative background pixels, and tocalculate, for all of missing pixels, a texture direction map indicatingtexture directions each corresponding to one of the missing pixels,based on locations of the representative background pixels and locationsof the missing pixels included in the missing pixel region; and afilling unit configured to entirely fill, at one time, the missing pixelregion with the background pixels without using the pixels of theforeground image region, while considering continuity at a boundary withthe background image region and without considering discontinuity at aboundary with the foreground image region, in accordance with thetexture direction map indicating the texture directions of the missingpixels.